Skip to main content

Azure Kubernetes Fleet Manager

Azure Kubernetes Fleet Manager is a service designed to provide unified management and orchestration for multiple Azure Kubernetes Service (AKS) clusters. It simplifies operations in multi-cluster environments and enables consistent configuration management and update strategies.

What is Fleet Manager​

Fleet Manager groups multiple AKS clusters into a "fleet" and provides the following capabilities:

  • Unified Management: Manage multiple clusters from a single control plane
  • Multi-cluster Kubernetes Resource Propagation: Efficiently deploy Kubernetes resources across multiple clusters
  • Staged Rollouts: Apply changes across clusters in stages
  • Unified Update Strategy: Manage Kubernetes version and configuration updates collectively

Key Components​

1. Fleet Resource​

A Fleet is a logical container that groups multiple AKS clusters together.

apiVersion: containerservice.azure.com/v1
kind: Fleet
metadata:
  name: production-fleet
spec:
  hubProfile:
    dnsPrefix: production-fleet-hub

2. Member Cluster​

Individual AKS clusters that join a Fleet are called member clusters.

# Join a cluster to the Fleet
az fleet member create \
  --resource-group myResourceGroup \
  --fleet-name production-fleet \
  --name cluster-east \
  --member-cluster-id /subscriptions/.../clusters/aks-east

3. Hub Cluster​

Fleet Manager optionally uses a Hub Cluster to provide unified management through the Kubernetes API.

Kubernetes Resource Propagation​

Fleet Manager enables efficient deployment of Kubernetes resources across multiple clusters.

ClusterResourcePlacement​

ClusterResourcePlacement defines which resources to deploy to which clusters.

apiVersion: placement.kubernetes-fleet.io/v1beta1
kind: ClusterResourcePlacement
metadata:
  name: deploy-frontend
spec:
  resourceSelectors:
    - group: ""
      version: v1
      kind: Namespace
      name: frontend
    - group: apps
      version: v1
      kind: Deployment
      labelSelector:
        matchLabels:
          app: frontend
  policy:
    placementType: PickAll

Placement Policies​

PickAll​

Deploys resources to all member clusters.

policy:
  placementType: PickAll

PickN​

Deploys to a specified number of clusters.

policy:
  placementType: PickN
  numberOfClusters: 3

PickFixed​

Deploys to specific named clusters.

policy:
  placementType: PickFixed
  clusterNames:
    - cluster-east
    - cluster-west

Cluster Selector​

Select clusters based on labels.

policy:
  placementType: PickAll
  affinity:
    clusterAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        clusterSelectorTerms:
          - labelSelector:
              matchLabels:
                environment: production
                region: japan

Staged Rollouts (Update Strategy)​

Fleet Manager provides powerful capabilities for applying changes in stages.

Update Run​

Use Update Run to execute cluster updates in stages.

apiVersion: fleet.azure.com/v1alpha1
kind: UpdateRun
metadata:
  name: k8s-upgrade-1-28
spec:
  updateStrategyId: production-strategy
  managedClusterUpdate:
    upgrade:
      type: Full
      kubernetesVersion: 1.28.5

Update Strategy​

Define update strategies to control the sequence and timing of rollouts.

apiVersion: fleet.azure.com/v1alpha1
kind: UpdateStrategy
metadata:
  name: production-strategy
spec:
  stages:
    - name: dev-stage
      groups:
        - name: dev-clusters
      afterStageWaitInSeconds: 3600
    - name: prod-stage
      groups:
        - name: prod-east-clusters
        - name: prod-west-clusters
      afterStageWaitInSeconds: 7200

Stages and Groups​

  • Stage: Sequential update phases
  • Group: Sets of clusters updated simultaneously
  • Wait Period: Waiting time between stages

Use Cases​

1. Multi-Region Deployment​

Deploy applications spanning multiple regions at once.

apiVersion: placement.kubernetes-fleet.io/v1beta1
kind: ClusterResourcePlacement
metadata:
  name: global-app-deployment
spec:
  resourceSelectors:
    - group: apps
      version: v1
      kind: Deployment
      name: my-app
  policy:
    placementType: PickAll
    affinity:
      clusterAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          clusterSelectorTerms:
            - labelSelector:
                matchExpressions:
                  - key: region
                    operator: In
                    values:
                      - japaneast
                      - japanwest
                      - eastus

2. Environment-based Cluster Management​

Efficiently manage development, staging, and production clusters.

# Development Fleet
az fleet create \
  --resource-group dev-rg \
  --name dev-fleet \
  --location japaneast

# Production Fleet
az fleet create \
  --resource-group prod-rg \
  --name prod-fleet \
  --location japaneast \
  --enable-hub

3. Kubernetes Version Upgrades​

Upgrade Kubernetes versions across all clusters in stages.

apiVersion: fleet.azure.com/v1alpha1
kind: UpdateRun
metadata:
  name: upgrade-to-1-29
spec:
  updateStrategyId: safe-upgrade-strategy
  managedClusterUpdate:
    upgrade:
      type: Full
      kubernetesVersion: 1.29.0
    nodeImageSelection:
      type: Latest

Best Practices​

1. Appropriate Grouping​

Logically group clusters and leverage labels.

# Apply labels to clusters
az aks update \
  --resource-group myResourceGroup \
  --name aks-cluster \
  --tags environment=production region=japaneast tier=frontend

2. Implement Staged Rollouts​

Always apply production changes in stages.

stages:
  - name: canary
    groups:
      - name: canary-cluster
  - name: production
    groups:
      - name: prod-clusters
    afterStageWaitInSeconds: 86400  # Wait 24 hours

3. Monitoring and Observability​

Monitor update success/failure at each stage.

# Check Update Run status
az fleet updaterun show \
  --resource-group myResourceGroup \
  --fleet-name production-fleet \
  --name upgrade-run-1

4. Rollback Strategy​

Define rollback procedures in advance for when issues occur.

apiVersion: fleet.azure.com/v1alpha1
kind: UpdateRun
metadata:
  name: rollback-to-1-28
spec:
  updateStrategyId: emergency-rollback
  managedClusterUpdate:
    upgrade:
      type: Full
      kubernetesVersion: 1.28.5

5. Validate Resource Propagation​

Verify resource placement before deployment.

# Check ClusterResourcePlacement status
kubectl get clusterresourceplacement deploy-frontend -o yaml

Fleet Manager and GitOps Integration​

Fleet Manager can be combined with GitOps tools (FluxCD, Argo CD).

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: fleet-config
  namespace: flux-system
spec:
  interval: 1m
  url: https://github.com/myorg/fleet-config
  ref:
    branch: main
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: fleet-placements
  namespace: flux-system
spec:
  interval: 10m
  sourceRef:
    kind: GitRepository
    name: fleet-config
  path: ./placements
  prune: true

Limitations​

  • Maximum of 100 member clusters per Fleet
  • Hub Cluster is deployed in a single region
  • Cross-cluster network communication is not automatically configured
  • Some Kubernetes resource types may not be propagatable

Summary​

Azure Kubernetes Fleet Manager significantly simplifies multi-cluster environment management. Features like unified management, multi-cluster deployments, and staged rollouts enable enterprise-grade Kubernetes operations.

By applying appropriate strategies and best practices, you can build scalable and reliable multi-cluster environments.

References​